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Catholic elementary schools must continue to invest in the professional develop¬ 
ment of math and science teachers in order to prepare students for the challeng¬ 
ing work that lies ahead of them. The purpose of the study was to examine the 
degree to which the Initiative for Catholic Schools (ICS), a 2-year professional 
development program for science and math teachers, demonstrated positive out¬ 
comes within the context of Catholic elementary education across the five levels 
of impact for a professional development program: participants ’reactions, par¬ 
ticipants ' learning, organization support and change, participants ’ use of new 
knowledge and skills, and student learning outcomes. The results provide evi¬ 
dence of positive outcomes in the participants 'reactions, participants 'learning, 
organization support and change, and participants ’ use of new knowledge and 
skills. The impact on student learning outcomes was less consistent and varied 
by grade level. 

T he United States places a high priority on science and mathematics ed¬ 
ucation to advance its position in a global society. Disappointingly, the 
progress made in reforming science and mathematics education has 
been mixed. According to the most recent Trends in International Mathematics 
and Science Study (TIMSS) comparing fourth and eighth graders’ achieve¬ 
ment to their peers in other countries, fourth grade students’ achievement 
improved considerably, but eighth grade achievement remained flat (Mullis, 
Martin, Gonzalez, & Chrostowski, 2004). Given that our nation’s youth com¬ 
pete in an increasingly global economy emphasizing math, science, and tech¬ 
nology advancement, science and mathematics educators must respond to 
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the renewed demands for quality science and mathematics instruction or risk 
leaving their students ill prepared to succeed. 

The Essential Role of Professional Development 

With the passage of the No Child Left Behind Act in 2001, public school dis¬ 
tricts are accountable for providing “highly qualified” teachers for every stu¬ 
dent. Public school districts are provided federal funding through Title II for 
the purposes of preparing, training, and recruiting high-quality teachers. In 
order for Catholic elementary schools to continue to attract, train, and recruit 
high-quality science and math teachers, considerations must be made for the 
professional development of these educators. An intentional, ongoing, and 
systemic model for professional development is critical for meeting the de¬ 
mands in today’s Catholic school classrooms. 

The purpose of this article is to describe the results of a comprehensive 
evaluation of the Initiative for Catholic Schools (ICS), a 2-year professional 
development program designed to increase the knowledge and skills of sci¬ 
ence and math teachers, strengthen leadership, and increase student achieve¬ 
ment in science and math in Catholic elementary schools. The evaluation 
examined the degree to which the ICS program demonstrated positive out¬ 
comes across the five critical levels of impact for a professional development 
program: participants’ reactions, participants’ learning, organization support 
and change, participants’ use of new knowledge and skills, and student learn¬ 
ing outcomes. In this article, a review of the literature on professional devel¬ 
opment is provided, followed by a detailed description of the ICS program, 
the methods used to evaluate the program, the results obtained, and a discus¬ 
sion of these results within the larger context of professional development 
and educational accountability. 

Five Levels of Impact for a 
Professional Development Program for Educators 

The current emphasis on professional development comes from the growing 
recognition that education is a dynamic, challenging professional field where 
the stakes are high (Guskey, 2000). Guskey proposed a model for evaluat¬ 
ing the impact of professional development that is comprised of five levels: 
(a) participants’ reactions, (b) participants’ learning, (c) organization support 
and change, (d) participants’ use of new knowledge and skills, and (e) student 
learning outcomes. Although designed as a model for evaluation, understand¬ 
ing which outcomes will be measured and how is an essential early step in 
program development (Worthen, Sanders, & Fitzpatrick, 2004). Recognition 
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of the potential impact of professional development on multiple levels chal¬ 
lenges those designing professional development programs to consider the 
linkages between the levels (e.g., how participants’ use of new knowledge 
and skills will realistically lead to improved student learning outcomes) in 
order to maximize comprehensive reform. 

Level 1: Participants’ Reactions 

Measuring participants’ initial satisfaction with a professional development 
program can provide immediate information to help improve design and de¬ 
livery (Guskey, 2000). Such information can also give insight into the de¬ 
gree to which the participants are grasping the content of the professional 
development program and its perceived value. Equally important, measuring 
participants’ reactions can also reveal procedures and activities that promote 
participants’ learning as well and those that inhibit learning (Guskey, 2000). 

Level 2: Participants’ Learning 

The degree to which participants learn what was intended is critical to estab¬ 
lishing the merit and worth of a professional development program. Specific 
criteria and indicators of learning must be defined early in the program de¬ 
velopment process (Guskey, 2000). In a review of studies investigating the 
effects of professional development on student learning, the greatest effects 
were observed when teachers were engaged with knowledge directly relevant 
to what students were learning (Kennedy, 1998 as cited in Guskey, 2000). 

In addition to the content knowledge presented in a professional develop¬ 
ment program, teachers can also be expected to gain in their own estimation 
of their ability to teach effectively. Self-efficacy is defined as a belief in one’s 
own abilities to perform an action or activity necessary to achieve a goal or 
task (Bandura, 1997). Beliefs about one’s own abilities to accomplish spe¬ 
cific tasks are powerful predictors of behavior. Self-efficacy beliefs influence 
choices, effort, and persistence in the face of adversity (Pajares, 1997). 

As it relates to teachers, teacher self-efficacy is the belief that one can 
bring about desired outcomes of student engagement and learning, even 
among those students who may be difficult or unmotivated (Tschannen-Moran 
& Hoy, 2001). Teacher self-efficacy has been linked to teachers’ taking re¬ 
sponsibility for student achievement (Guskey, 1982, 1988) and greater persis¬ 
tence in working with struggling students (Gibson & Dembo, 1984). Teacher 
self-efficacy is predictive of a willingness to implement innovative teaching 
strategies and improve methods of instruction (Allinder, 1994; Guskey, 1984, 
1988; Smylie, 1988; Stein & Wang, 1988). 
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A teacher’s self-efficacy has been found to play a critical role in his or 
her ability to impact student achievement (Ashton, 1985; Ashton & Webb, 
1986). With particular respect to science and math education, several studies 
have shown a positive association between teacher self-efficacy and elemen¬ 
tary students’ achievement in science (Cannon & Scharmann, 1996), student 
motivation in mathematics for students transitioning to junior high school 
(Midgley, Feldlaufer, & Eccles, 1989), and achievement in computer technol¬ 
ogy (Ross, Hogaboam-Gray, & Hannay, 2001). 

Level 3: Organization Support and Change 

Organizations have a powerful influence on all aspects of professional devel¬ 
opment (Guskey, 2000). Successes attained with individual aspects of profes¬ 
sional development can be stifled, halted, or essentially canceled by seemingly 
immutable factors in the organization’s culture (Fullan, 1993; Sparks, 1996; 
Sparks & Flirsh, 1997). For professional development programs to maximize 
their success, careful consideration must be given to organizational elements 
that include organizational policies (aligned with organization’s mission), 
resources, protection from intrusions, openness to experimentation and al¬ 
leviation of fears, collegial support, principals’ leadership and support, high¬ 
er-level administrators’ leadership and support, recognition of success, and 
provision of time (Guskey, 2000). 

Level 4: Participants ’ Use of New Knowledge and Skills 

The degree to which teachers are able and willing to apply new knowledge 
and implement new instructional strategies competently is the most criti¬ 
cal measure of the effectiveness of a professional development program. 
Producing deep, meaningful, and sustainable changes in teachers’ instruc¬ 
tional practices is difficult, according to Coburn (2003), as teachers are: (a) 
likely to gravitate toward approaches that are congruent with their prior prac¬ 
tices and avoid approaches that conflict with prior practices (Spillane, 2000), 
(b) focus on surface manifestation (such as discrete activities, materials, or 
classroom organization) rather than deeper pedagogical principles (Cobum, 
2002; Spillane, 2000; Spillane & Callahan, 2000; Spillane & Zeuli, 1999), 
and (c) graft new approaches on top of existing practices without altering 
classroom norms and routines (Cobum, 2002; Cuban, 1993). Incorporating 
new practices and techniques to unique, on-the-job conditions is an uneven 
process that requires time and effort (Joyce & Showers, 1995). The obstacles 
to increasing teachers’ use of new knowledge and skills highlight the chal- 
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lenges of planning, implementing, and evaluating high-quality professional 
development for teachers. 

Level 5: Student Learning Outcomes 

The ultimate goal of any professional development program for teachers is an 
increase in student achievement. The results of a recent study provide addi¬ 
tional evidence that student achievement is largely attributable to the contri¬ 
bution of the teacher. Using data from a 4-year experiment in which teachers 
and students were randomly assigned to classes to estimate teacher effects on 
student achievement, Nye, Konstantopoulos, and Hedges (2004) found that 
the variance due to the differences among teachers was substantial in compar¬ 
ison to the variance attributable to naturally occurring school effects. Thus, 
“which teacher a student happens to get within a school matters more than 
which school the student happens to attend” (p. 247). Empirical evidence of 
the impact of teacher effects on student achievement supports previous re¬ 
search concluding that the teacher is the primary school-based determinant 
who affects the variance in student achievement (Goldhaber & Brewer, 1997; 
Sanders & Rivers, 1996). 

The Initiative for Catholic Schools (ICS) Program 

The ICS program was a 2-year professional development program designed 
to improve teaching strategies and content knowledge in science and mathe¬ 
matics and school leadership. The goals of the ICS program are: (a) to devel¬ 
op and strengthen leadership in Catholic elementary schools, (b) to improve 
the curriculum in science and mathematics, (c) to improve student learning 
and retention in basic science and mathematics content, and (d) to implement 
a student-oriented pedagogy. Seventy-seven Catholic elementary schools lo¬ 
cated in a Midwestern city were invited to submit a proposal for participation 
in the program. Twenty-one school teams were selected based on their re¬ 
sponses to questions pertaining to their goals and intentions as they related to 
science and mathematics. Each school team consisted of at least one science 
teacher, at least one mathematics teacher, and the school principal. The par¬ 
ticipating schools were diverse in socioeconomic status, ethnic backgrounds, 
and location (see Appendix A). 

The participants in the 2-year program met for monthly workshops during 
the academic year and for 2 weeks in each summer session. Science and math 
educators met as separate groups. The monthly sessions and the two 5-day 
summer sessions were taught by university faculty members whose areas of 
expertise were science and mathematics education. The monthly workshops 
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examined topics that included constructivist learning theory; the learning cy¬ 
cle; national, state, and district standards; and pedagogical practices such as 
assessment, inquiry-based instructional techniques, curriculum planning and 
the use of technology to enhance instruction. The focus of the summer ses¬ 
sions was content, although effective teaching strategies were modeled and 
discussed. In the summer of the first year, science educators focused on life 
science and math educators examined content in geometry and probability. In 
the summer of the second year science educators examined physical science 
content and math educators explored algebraic logic and measurement. 

Guskey’s (2000) model provided a framework for the design of the ICS 
program. At Level 1, participants’ reactions were sought, as science and math 
teachers were asked to submit written input on the daily summer sessions at 
the completion of each day and asked to provide a written reflection at the 
completion of the 5-day workshop. Standard course evaluations developed by 
the host university were also used to assess participants’ reactions. 

At Level 2, participants’ learning was evaluated using multiple methods 
throughout the professional development program. First, graduate-level cred¬ 
it was awarded to the ICS teacher participants and grades were assigned in ac¬ 
cordance with the university guidelines. In addition, pre- and post-tests were 
administered to the teachers during the summer sessions to measure the extent 
of science and mathematics knowledge acquisition. Lesson plans were also 
collected and the teachers were directly observed teaching their students in 
their school buildings. Math teachers designed their curriculum based on the 
course of study through the development of a curriculum map as further evi¬ 
dence of development in their pedagogical knowledge and skills. A pre- and 
post-test measure of science teaching self-efficacy and math teaching self- 
efficacy was also administered to science and math educators, respectively. 

Organization support and change (Level 3) was addressed in the design 
of the ICS program, with each participating school committing a team of 
educators and administrators to the 2-year professional development process. 
Furthermore, one of the objectives of the ICS program was to develop teacher 
leaders at each school. Teacher leaders were expected to share their knowl¬ 
edge and skills with other science and mathematics teachers in their build¬ 
ings. The principal was critical in establishing a venue where sharing could 
occur and in supporting the implementation of new teaching strategies. In 
addition, each school team was required to develop a site-based school im¬ 
provement project in science or mathematics in their schools. These projects 
ranged from executing a science enrichment program to revising and imple¬ 
menting a mathematics curriculum. 
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As evidence of participants’ use of new knowledge and skills (Level 4), 
science and math teachers were required to submit their most recent lesson 
plan on two occasions during the ICS program year. The lesson descriptions 
were analyzed by university faculty based on the presence of critical elements 
of constructivist learning theory. A list of these critical elements appears in 
the Method section. 

Student learning outcomes (Level 5) were assessed using standards-based 
tests of life science, probability, and geometry. These tests were administered 
at the beginning and end of the school year to measure achievement gain. 
These same tests were administered to students attending one of three com¬ 
parison schools in order to determine the degree to which the gains achieved 
by the ICS students met or exceeded the gains demonstrated by students 
whose teachers did not participate in the ICS program. 

Method 

Purpose of the Current Study 

The purpose of the study was to examine the degree to which the ICS program 
demonstrated positive outcomes across the five levels of impact for a profes¬ 
sional development program: participants’ reactions, participants’ learning, 
organization support and change, participants’ use of new knowledge and 
skills, and student learning outcomes. This investigation extends the previous 
research by rigorously examining the impact of a comprehensive professional 
development program within the context of Catholic elementary education in 
general, and math and science education, specifically. A recent review of the 
research literature on professional development programs for teachers indi¬ 
cated that very few empirical studies of the impact of these programs existed 
(Mohler, Morrison, Hunley, & Grogan, 2007). Of these studies, only a minor¬ 
ity included a clearly stated goal, a clear description of the methodology, and 
specific outcome measures. 

Setting 

Seventy-seven Catholic elementary schools located in a Midwestern city were 
invited to submit a proposal for participation in the ICS program. Of these po¬ 
tential participants, 21 Catholic elementary schools submitted a proposal and 
were then selected for participation. A team from each school consisted of 
at least one science teacher, at least one mathematics teacher, and the school 
principal. The participating schools were diverse in socioeconomic status, 
ethnic backgrounds, and location (see Appendix A). All funding for ICS was 
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granted by a private foundation, the Buenger Foundation, in collaboration 
with a Catholic university in the area. 

Participants 

Science educators. Twenty-four science teachers participated in the ICS 
program. Among these teachers there were 23 females and 1 male. The prior 
teaching experience of the science teachers ranged from 0-39 years, with an 
average of 14 years of teaching experience (an average of 9 years teaching 
science). Due to attrition, there were 18 science teachers participating in the 
ICS program as of the second year. 

Math educators. The math educators in this study were comprised of 24 
math teachers from the 21 schools participating in the ICS program. All of the 
math teachers were female. The prior teaching experience of the math teach¬ 
ers ranged from 0-34 years, with an average of 13 years of teaching experi¬ 
ence (an average of 12 years teaching math). Through attrition, there were 22 
math teachers participating in the ICS program at the end of the second year. 

Design and Procedures 

The study used descriptive research methods to describe quantitative out¬ 
comes of the ICS program. Pre-experimental within-subjects designs were 
used to examine changes in mean ratings of teacher self-efficacy and mean 
gains in student achievement over the course of the ICS program. Changes 
in teacher self-efficacy ratings were measured across a 10-month period from 
the beginning of the ICS program to midway through the program. The teach¬ 
er participants were provided a paper version of the teacher self-efficacy rat¬ 
ing scale and a self-addressed, stamped envelope. 

A pre-test post-test comparison group quasi-experimental design was 
used to examine the gains in science and math achievement relative to those 
of a comparison group. Tests of student achievement in science and math 
were administered in October (pre-test) and again in April (post-test) of the 
same academic year to students in the ICS schools at the grade levels taught 
by ICS teacher participants. These same tests were administered in the same 
time frame to students in three Catholic elementary schools from the same 
school system. The comparison schools were selected on the basis that they 
featured a range of socioeconomic diversity among the student population 
that was comparable to that of the ICS schools (see Appendix A). 
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Measures and Analyses 

Science Teaching Efficacy Beliefs Instrument (STEBI). The STEBI, de¬ 
veloped by Riggs and Enochs (1990), was used to measure the teachers’judg¬ 
ment of their self-efficacy in teaching science. This instrument measures two 
aspects of science teacher efficacy. The personal science teaching efficacy 
(PSTE) assesses teachers’ degrees of confidence that they can perform a given 
action successfully. The science teaching outcome expectancy (STOE) assess¬ 
es teachers’ perceptions that a given action will have a favorable result. The 
PSTE subscale consisted of 13 items and the STOE subscale was comprised 
of 10 items. The STEBI was structured using a 5-point Likert rating scale, 
where 5 = “Strongly Agree,” 4 = “Agree,” 3 = “Uncertain,” 2 = “Disagree,” 
and 1 = “Strongly Disagree.” An analysis of the internal consistency of the 
STEBI provides support for the basic integrity of the two subscales and the 
overall reliability of the instrument (Bleicher, 2004). 

Changes in teachers’ self-efficacy were analyzed by comparing the teach¬ 
ers’ self-ratings at the beginning of the ICS program with the teachers’ self- 
ratings midway through the ICS program (10 months later), a critical point in 
the formative evaluation of the program as determined by the program’s key 
stakeholders and sponsors. Paired-sample t-tests were conducted to determine 
the degree to which the changes in the mean self-ratings were statistically 
significant. Effect sizes were calculated to determine the strength of the mean 
change, where an effect size of .20-.49 represented a small effect, .50-.79 in¬ 
dicated a medium effect, and .80 or greater represented a large effect. 

Mathematics Teaching Efficacy Beliefs Instrument (MTEBI). The STEBI 
was adapted to create the Mathematics Teaching Efficacy Belief Instrument 
by university faculty members in science education and math education di¬ 
rectly involved in the ICS program for the purpose of this evaluation. Changes 
in teachers’ self-efficacy were analyzed by comparing the teachers’ self-rat¬ 
ings at the beginning of the ICS program with the teachers’ self-ratings mid¬ 
way through the ICS program (10 months later). Paired-sample t-tests were 
conducted to determine the degree to which the changes in the mean self- 
ratings were statistically significant. Effect sizes were calculated to determine 
the strength of the mean change, where an effect size of .20-.49 represented a 
small effect, .50-.79 indicated a medium effect, and .80 or greater represented 
a large effect. 

Assessment of lesson plans. ICS teachers were required to submit their 
most recent lesson plan at the beginning of the program and again at the 
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end of the program. Lesson plans were assessed by university faculty in sci¬ 
ence and math education using a rubric comprised of the critical elements 
of constructivist learning theory: student directed, conceptual development, 
performance-based objectives, use of science equipment/math manipulatives, 
group work, and accurate science/math content. The rubric for assessing 
science lesson plans also included inquiry approach and the rubric for as¬ 
sessing math lesson plans included use of technology, multiple solutions, and 
proper terminology. 

Site-based improvement project evaluation. As previously described, each 
school team participating in the ICS was required to develop a site-based school 
improvement project related to improving the math and/or science program in 
their school. The plans were evaluated by the ICS project director and two uni¬ 
versity faculty members directly involved with the ICS program. Judgments of 
the quality of the plans were based on three factors: comprehensiveness, feasi¬ 
bility, and likelihood of attaining the desired objective. The quality of the plans 
were rated using a 5-point categorical rating scale, where 5 = “Very high qual¬ 
ity,” 4 = “High quality,” 3 = “Average,” 2 = “Below average,” and 1 = “Poor.” 
The degree to which the plans aligned with the goals of the ICS program was 
assessed using a 5-point categorical rating scale, where 5 = “Perfectly aligned,” 
4 = “Mostly aligned,” 3 = “Moderately aligned,” 2 = “Minimally aligned,” and 
1 = “Not aligned” (see Appendix B). 

Standards-based tests of achievement in math and science. University 
faculty members with expertise in science and math education developed 
the Standards-Based Tests of Achievement in Math and Science. Test items 
were sampled from items on the Proficiency Test at Grades 4 and 6, from the 
Ohio Achievement Test for Grade 8, and from the Ohio Diagnostic Test for 
Grades 1, 2, and 3. This test content was available in the form of sample test 
questions from the state Department of Education website (Ohio Department 
of Education, 2008). Questions were selected based on their teach-test con¬ 
tent overlap with the ICS program (see Table 1). The teachers learned the 
same concepts albeit at the collegiate level during the summer sessions. In 
this way, there were links connecting teachers’ content knowledge, the cur¬ 
riculum established by the Archdiocese, and the students’ experiences in the 
classroom. The teachers who participated in the ICS program administered 
the Standards-Based Test of Achievement in Science (Grades 4-6) and Math 
(Grades 1-3) to their students in October and again in April of the same aca¬ 
demic year. This academic year represented the ICS teachers’ second year of 
participation in the ICS program. 
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Results 

Level 1: Participants’ Reactions 

According to the data gathered from the course evaluations, all but one of the 
teacher participants responded that the ICS course met or exceeded their ex¬ 
pectations. Their qualitative comments indicated that they valued being able 
to visualize science and math concepts in a conceptual manner. All but one of 
the teachers judged the content and pedagogy emphasized in the ICS course 
as being beneficial (the exception was a first grade teacher who commented 
that the math content geared to Grades 1 -3 was too advanced for her students). 
The qualitative comments centered on deepening the conceptual knowledge 
of concepts, connecting the concepts to appropriate hands-on materials, and 
acquiring many activities they could use in the classroom. Likewise, when 
asked which components of the ICS course were the most beneficial, teachers 
responded that they valued the materials, handouts, and activities they could 
use in the classroom. According to these participants, math activities ad¬ 
dressing more advanced concepts (i.e., beyond the third grade) were the least 
beneficial aspects of the ICS course. 

Level 2: Participants’ Learning 

Grades earned in graduate-level courses. In science, 17 of the partici¬ 
pants received an A and 1 participant received a B for both the life science 
and physical science courses. In the probability and geometry course, 20 
participants received an A, 2 participants received a B and 2 participants re¬ 
ceived a C. For the algebraic thinking and measurement course, 21 partici¬ 
pants received an A, 1 participant audited the class and received an S. One 
math teacher withdrew from the course due to illness. 

Included in the course grades were the teachers’ scores on the pre- and 
post-test on content knowledge. The mean change in performance, as given 
by the difference in the percentage correct from the pre- to the post-test of the 
Teachers’ Content Knowledge, was as follows: Probability (+48.6 percent¬ 
age points), Geometry (+29.2 percentage points), Algebra (+45.3 percentage 
points), Measurement (+46.1 percentage points). Life Sciences I (+39.6 per¬ 
centage points), Life Sciences II (+14.2 percentage points), Physical Sciences I 
(+25.0 percentage points), and Physical Sciences II (+23.8 percentage points). 
The change in percentage correct from pre-test to post-test was statistically 
significant at the .05 level for all of the gain scores listed above. 
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Table 1 


Standards-Based Tests of Achievement in Math and Science Test Construction 


Grade Level 

Content Assessed 

Number of Items and Item Format 

Grade 1 

Geometry 

7 Fill-in/Open-Ended 

2 Multiple Choice 

1 Extended Response 


Probability 

9 Fill-in/Open-Ended 

4 Multiple Choice 

Grade 2 

Geometry 

7 Fill-in/Open-Ended 

2 Multiple Choice 

1 Extended Response 


Probability 

8 Fill-in/Open-Ended 

2 Multiple Choice 

Grade 3 

Geometry 

16 Fill-in/Open-Ended 


Probability 

14 Fill-in/Open-Ended 

Grade 4 

Life Science 

8 Multiple Choice 

2 Extended Response 

Grade 5 

Life Science 

8 Multiple Choice 

2 Extended Response 

Grade 6 

Life Science 

9 Multiple Choice 

1 Extended Response 


Teachers’ self-efficacy outcomes. The results of this investigation indicate 
that science and mathematics teachers participating in the ICS program dem¬ 
onstrated increases in their teaching self-efficacy during the course of their 
participation in the ICS program. Science teachers’ ratings increased overall 
from a mean of 88.53 (SD = 9.52) to 98.53 (SD= 7.21). Thi s increase in science 
teachers’ self-reported self-efficacy was statistically significant (t = -4.79, 14, 
p = 0.00) and represented a strong effect (ES = 1.05). 

Similar results were obtained for the mathematics teachers participating 
in the ICS program. Mathematics teachers’ ratings increased overall from 
a mean of 88.33 ( SD = 8.38) to 95.07 ( SD = 7.97). Thi s increase in math 
teachers’ self-reported self-efficacy was statistically significant (t = -5.43, 14, 
p = 0.00) and represented a strong effect (ES = .80). 

Level 3: Organization Support and Change 

All of the schools participating in the ICS program committed a team of edu¬ 
cators and administrators to the initiative. One hundred percent of the schools 
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completed a site-based school improvement project in science or mathemat¬ 
ics in their schools. A list of the projects along with a rating of their quality 
and alignment with the goals of the ICS program appears in Appendix B. 

Level 4: Participants ’ Use of New Knowledge and Skills 

Fifteen math teachers submitted a brief description of their most recent math 
lesson at the beginning of the ICS program (January) and at the end of the 
calendar year (December) for analyses by a university faculty member. The 
lesson descriptions were coded based on the presence of critical elements of 
constructivist learning theory. Analyses of these lesson plans indicate that 
math teachers demonstrated marked increases in the application of effective 
math pedagogy as evidenced by their descriptions of their most recent lessons 
(see Figure 1). 

Nineteen science teachers submitted a brief description of their most re¬ 
cent science lesson at the beginning of the ICS program (January) and again 
at the end of the ICS program (August of the following year) for analyses by 
a university faculty member. The lesson descriptions were coded based on 
the presence of critical elements of constructivist learning theory. Science 
teachers participating in the ICS program demonstrated marked increases in 
the application of effective science pedagogy as evidenced by their descrip¬ 
tions of their most recent lessons (see Figure 2). 



Figure 1. Analyses of math teachers’ lesson plans, January to December. 
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Figure 2. Analyses of science teachers’ lesson plans, January to August. 


Level 5: Student Learning Outcomes 

During the second year of program participation, students taught by an ICS 
teacher demonstrated gains in math achievement in probability that were sta¬ 
tistically significantly greater than those of the students in the comparison 
schools at Grade 3 (t= 5.805, 263, p = 0.00). There were no statistically sig¬ 
nificant differences between the ICS students and the comparison school stu¬ 
dents at Grades 1 and 2 (see Table 2). In geometry, students taught by an ICS 
teacher demonstrated gains in math achievement that were statistically signif¬ 
icantly greater than those of the students in the comparison schools at Grade 
3 (t= 7.281, 263 ,p = 0.00). There were no statistically significant differences 
between the ICS students and the comparison school students at Grades 1 and 
2 (see Table 3). 

During the second year of program participation, students taught by an 
ICS teacher demonstrated gains in science achievement that were statistically 
significantly greater than those of the students in the comparison schools in 
life science at Grade 5 (t = 5.570, 325, p = 0.00) and Grade 6 (t = 2.57, 306, 
p = 0.01). Students in the comparison schools outperformed ICS students at 
Grade 4 (see Table 4). 
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Table 2 


Results of the Standards-Based Math Test in Probability 



Number of 
Students 

October 2005 
Mean ( SD ) 

May 2006 
Mean (SD) 

Achievement 

Gain 

Grade 1 

ICS Students 

22 

69.9% 

(17.21) 

89.5% 

(12.46) 

+19.6 

Comparison 

96 

65.4% 

(17.33) 

81.0% 

(19.68) 

+15.5 

Grade 2 

ICS Students 

44 

49.1% 

(14.91) 

69.6% 

(23.12) 

+20.5 

Comparison 

55 

58.7% 

(14.41) 

73.6% 

(14.58) 

+14.9 

Grade 3 

ICS Students 

164 

47.0% 

(17.27) 

70.8% 

(17.55) 

+23.8** 

Comparison 

101 

51.9% 

(19.20) 

61.2% 

(18.14) 

+9.3 

** Statistically significant at .01 level 




Table 3 

Results of the Standards-Based Math Test in Geometry 




Number of 
Students 

October 2005 
Mean (SD) 

May 2006 
Mean (SD) 

Achievement 

Gain 

Grade 1 

ICS Students 

22 

70.0% 

(21.11) 

88.0% 

(14.41) 

+18.0 

Comparison 

96 

56.2% 

(17.97) 

80.9% 

(15.35) 

+24.7 

Grade 2 

ICS Students 

44 

45.9% 

(15.90) 

57.5% 

(15.27) 

+15.8 

Comparison 

55 

55.8% 

(14.87) 

62.7% 

(14.71) 

+21.2 

Grade 3 

ICS Students 

164 

14.5% 

(7.87) 

44.2% 

(19.58) 

+29.7** 

Comparison 

101 

16.5% 

(9.95) 

29.5% 

(15.23) 

+13.0 


** Statistically significant at .01 level 
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Table 4 

Results of the Standards-Based Science Test in Life Science 



Number of 
Students 

October 2005 
Mean (SD) 

May 2006 
Mean (SD) 

Achievement 

Gain 

Grade 4 

ICS Students 

64 

56.9% 

(20.15) 

66.3% 

(15.38) 

+9.4 

Comparison 

107 

46.5% 

(16.44) 

64.7% 

(16.50) 

+18.2* 

Grade 5 

ICS Students 

229 

49.2% 

(16.46) 

60.3% 

(16.82) 

+11.1* 

Comparison 

98 

63.3% 

(16.67) 

61.8% 

(15.36) 

-1.4 

Grade 6 

ICS Students 

197 

48.6% 

(18.04) 

61.5% 

(19.71) 

+12.9* 

Comparison 

111 

56.2% 

(22.36) 

63.0% 

(22.51) 

+6.8 


Statistically significant at .05 level 


Discussion 

The current climate in education, with its emphasis on accountability and 
highly qualified teachers, is supported by the notion that the professional de¬ 
velopment of teachers is critical to improving student learning. More than 
20 years of research suggests that notable improvements in education almost 
never take place in the absence of professional development (Guskey, 2000). 
Given that professional development efforts vary widely based on differenc¬ 
es in program context, the structure and format of the experience (process), 
and the context in which implementation occurs, the most relevant question 
when evaluating the impact of a professional development program is, “un¬ 
der what conditions (that is, what content, types of formats, contextual char¬ 
acteristics, and so forth) is professional development likely to yield positive 
effects?” (p. 33). 

Three key features of this professional development program provide in¬ 
sight into the conditions likely to yield positive effects for those seeking to 
replicate the ICS program. First, science and math teachers were recruited 
to participate as a team along with their principal. The rationale for using 
teams of educators is that different members of the team can provide dif¬ 
ferent expertise, support one another when new strategies and information 
are communicated, serve as models for their colleagues, and demonstrate 








Professional Development Model for Math and Science 491 


collaboration (Gideon, 1997; Joyner, 1997; Loucks-Horsley, Love, Stiles, 
Mundry, & Hewson, 2003). It is likely that the reasonably high retention rate 
among the participants of the ICS program was due to the team-based struc¬ 
ture, as quitting a professional development program is more difficult when 
it also involves quitting a team of colleagues who share a personal and pro¬ 
fessional connection. Including the principal as a member of the team was 
also critical for ensuring ongoing administrative support for the application 
of content knowledge and skills across settings (training center to classroom) 
and over time. 

A second key feature of the ICS program that created conditions likely 
to yield positive effects was the use of sufficiently strong incentives. Science 
and math teachers earned (tuition-free) graduate credit hours as a result of 
their participation in the ICS program and release time for the monthly work¬ 
shops during the school day. This second condition was judged to be particu¬ 
larly valued as teachers were not required to sacrifice their personal time in 
the evenings and on weekends in order to advance their knowledge and skills. 
In addition, science and math teachers were offered a modest stipend for their 
attendance at the monthly workshops and summer sessions, however, the sti¬ 
pend was judged by the key stakeholders of the ICS program to be a less val¬ 
ued incentive, and possibly superfluous. 

The third and final feature of the ICS program that helped set the condi¬ 
tions for positive outcomes was the fact that the participants were volunteers, 
rather than mandated attendees. In order to be accepted as a participant in the 
ICS program, teams of educators had to work together to develop a proposal 
describing their interest in the program and the potential of the ICS program 
to enhance their school’s effectiveness. Thus, the ICS program capitalized 
on a selection bias by including only individuals who demonstrated an inter¬ 
est and willingness to join a team of educators committed to the goals of the 
ICS program. 


Limitations 

There are several limitations inherent in this study that warrant attention in 
interpreting the results of this study. First, the teacher participants in the pro¬ 
gram were self-selected, as previously discussed, and may not be representa¬ 
tive of all math and science teachers in Catholic schools (selection bias). The 
second limitation is the small sample size of teachers participating in this 
study, which increases the possibility that any one participant could have a 
considerable impact on the teacher outcomes. A third limitation is the loss of 
participants during the program year (attrition). Although the attrition rate 
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for the program was judged to be typical for a multiyear program, the degree 
to which the program produced positive outcomes for all participants, includ¬ 
ing those who withdrew from the program, is unknown. A fourth limitation is 
the reliance on outcome measures that are based on self-report to gather data. 
The final limitation is the absence of a randomized controlled trial to compare 
outcomes for the program participants with those of a control group. In the 
absence of a randomized controlled trail, causation (i.e., that the program was 
the cause of the outcomes observed) cannot be established. 

Implications 

In the current climate of educational accountability and the demand for “highly 
qualified teachers” set forth by the No Child Left Behind Act, many states have 
instituted new regulations requiring ongoing professional development for 
teachers. Catholic schools and public schools alike seek to encourage teachers 
to explore new ideas and stay current with developments in the teaching pro¬ 
fession. The results of this study indicate that intensive, ongoing professional 
development can have a positive impact on teachers’ self-efficacy and on stu¬ 
dent achievement in science and mathematics in Catholic elementary schools. 
Further assessments to determine the effect of the professional development 
after the ICS program has ended will allow researchers to determine if the 
positive outcomes for teachers and students maintain, increase, or decrease 
over time. 
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Appendix B 

Site-Based School Improvement Projects: Mean Ratings and Standard Deviations 


Scale for Quality of the Plan 
5 = Very High 
4 = High 
3 = Average 
2 = Below Average 
1 = Poor 


Scale for Alignment with ICS Goals 
5 = Perfectly Aligned 
4 = Mostly Aligned 
3 = Moderately Aligned 
2 = Minimally Aligned 
1 = Not Aligned At All 


Project Title and Objective 

Quality 
of Plan 

Alignment 
with ICS 
Goals 

Creating an Enrichment Club for Mathematics and Science 

4.3 

5.0 

Objective: To think of science and mathematics as fun. 

(0.6) 

(0.0) 

Keyboarding Competence 

3.3 

2.7 

Objective: Students in K-8 will develop age appropriate 
keyboarding skills. 

(0.6) 

(1.5) 

Family Night for Mathematics and Science 

5.0 

4.1 

Objective: Develop a program for parental involvement 
where students teach their parents mathematics and science 
concepts they are learning. 

(0.0) 

(0.6) 

Creating a site based management system of governance using the 

4.0 

4.0 

power cycle. 

Objective: Using the power cycle, evaluate current 
practices to identify areas of concern, make 
recommendations to decision makers, and facilitate 
implementation. 

(0.0) 

(0.0) 

Algebra for All 

4.7 

5.0 

Objective: To revise the mathematics curriculum, 
instruction, and assessment with an emphasis on the 
algebra strand. 

(0.6) 

(0.0) 

Increasing student achievement and confidence in mathematics 

4.0 

4.7 

Objective: To have students value, enjoy, and become 
more confident in mathematics. 

(0.0) 

(0.6) 

Taking Community Service to a Higher Level. 

4.3 

4.3 

Objective: To increase student involvement in community 
service activities. 

(0.6) 

( 0.6) 

Increasing Students’ Terra Nova scores in Language Arts 

3.7 

3.0 

Objective: To increase Terra Nova scores in Language 

Arts. 

(0.6) 

(1.0) 

Establish a Resource Center for Science and Mathematics 

4.3 

4.3 

Instruction 

Objective: Increase the use of manipulatives and 
technology in classroom instruction. 

(1.2) 

(1.2) 
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Build Parish/School Community 

Objective: To make the school and parish a more unified 
community. 

4.0 

(0.0) 

3.7 

(0.6) 

Improved Instruction using Technology 

4.0 

4.7 

Objective: To expand the use of technology in 
mathematics and science instruction. 

(JO) 

(0.6) 

Manipulatives in the Mathematics and Science Classroom 

4.3 

4.7 

Objective: To expose students to strategies for learning 
mathematics and science through the use of manipulatives. 

(0.6) 

(0.6) 

Excellence in Mathematics and Science 

4.3 

4.3 

Objective: To improve student, teacher, and parent 
attitudes toward mathematics and science, methods, 
curriculum, and manipulatives. 

(0.6) 

(0.6) 

Improving student achievement in Geometry 

4.0 

4.3 

Objective: Improving student achievement in geometry. 

(0.0) 

(0.6) 

Reach for the Stars in Mathematics 

4.0 

4.3 

Objective: To increase curricular content and student 
interest in mathematics. 

(0.0) 

(0.6) 

Advanced Science and Mathematics Project 

4.7 

4.7 

Objective: To increase the opportunities for students to 
learn science and mathematics through manipulatives, 
hands-on experiments, and demonstrations. 

(0.6) 

(0.6) 

A Feasibility Study for a Special Education Program 

4.0 

4.0 

Objective: To increase educational opportunities for 
students in the parish with special needs. 

(1.0) 

(0.0) 

Enhancing Education through Improved Communication with 

4.3 

4.3 

Technology 

Objective: To improve communication between the school 
and its community. 

(0.6) 

(0.6) 













